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•  Verification 

-  Is  the  mathematical  model  solved  correctly? 

•  Validation 

-  Is  a  model  adequate  in  representing  the  “real”  physical  system? 

•  Contextual  process 

•  Subjective  results 

•  Various  types  of  model  outputs 


•  Accreditation 

-  Can  a  model  be  exercised  within  a  well-defined  scope? 
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Models  of  Dynamic  Systems 


•  Entire  time  history 
matters,  not  just  some 
of  its  features 


Single  Cell  -  ElectroThermal  Model 


Thermal  Model 

-  Two  States  (Tc,  Ts) 

-  Two  Inputs  (Tf,  Vin) 


Inputs 

-  I  (Current) 

-  Tf  (Inlet  Air  Temp) 

-  Vin  (Inlet  Flow  Velocity) 

Outputs 

-  Tc  (Core  Temperature  (°C)) 

-  Ts  (Surface  Temperature  (°C)) 

-  Vt  (Terminal  Voltage) 

-  SOC  (State  of  Charge) 


Ti.  V 


PI  C.  Park,  et  al.  20 
P]  X  Lin.  et  al.  201 1 
[7]  X  Lin,  et  al.  201 1 

Electrical  Model:  OCV-R-RC 

-  Three  States  (SOC,  VI,  V2) 

-  One  Input  (I) 

B,  rWn  rWV 

Voc'f  '  >  Cl 


<3 


[2]  Chen  et  al.  2006 

[3]  K.  Smith,  et  aL  2010 

[4]  Y.  Hu.  et  al.  2011 


Multiple,  time- 
dependent  output 


UAC  Experimental  vs.  Simulation  Result 
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•  Classification  of  validation  approaches 

•  Bayesian  interval  hypothesis  testing 

-  Quantifying  model  confidence 

-  Distribution-free  approach  by  means  of  bootstrapping 

-  Statistical  power  superiority 

•  Validation  benchmark  problem 

•  ARC-developed  electro-thermal  battery  model  validation 
for  energy  &  power  community  of  interest  application 
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Attributes  of  validation 
methodologies 
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MODELING  rno  SIMULATION,  TESTING  rno  VALIDATION 


Applicable  to  scalar  data 

Applicable  to  vector  data 

Applicable  to  scalar  time  series 

Applicable  to  vector  time  series 

Consider  multivariate  correlation 

Include  objective  criteria 

Quantify  model  confidence 

Can  incorporate  SME  opinions 

Can  work  without  normality  assumption 

Insensitive  to  type- 1  error 

Low  computational  cost 

Sample  size  independence 


l«chig»Cta 
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Bayesian  hypothesis  testing 
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Zhan,  Fu,  Yang,  Peng 
(2011) 
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Norm-based 
integration  bounds 
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MODELING  rho  SIMULATION,  TESTING  rno  VALIDATION 


•  Sensitivity  of  model  confidence 


K  = 


1 

J-s  \2na2 


exp 


n  +  1  / 
—2^ 


n 

71  +  1 


•  Based  on  intended  application/ 
SME  opinions 

e  =  fc||t|L 


•  Fit  in  the  Bayesian  model 
validation  framework 

c  =  a&s(M_1WTe) 


Example  of  norm-based 
integration  bounds 
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Variability-based 
integration  bounds 


•  What  confidence  should 
one  expect  when 
comparing  test  data 
with  themselves? 

•  Based  on  standard 
deviation  of  the  reduced 
test  data 

e  =  b’JdiagfX  t) 
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MODELING  rho  SIMULRTION.  TESTING  rno  VRLIDRTION 


99% 


b  (%) 


Example  of  variability-based 
integration  bounds 
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Bootstrap  method 


Distribution-free 

Ease  of  implementation 

Proven  asymptotic 
accuracy  of  inference 

Can  handle  non-i.i.d. 
data 

Can  handle  small 
sample  size 
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Sandia  thermal  benchmark 
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q  —  0 
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x  =  L 


Test 


CAE 


Validation 

Experiments 


T(x,  t)  =  Ta  +  — 

K 


X  1 

~L  +  2 


L  (cm) 


K.  J.  Dowding  etal.,  “Formulation  of  the  thermal  problem,”  CMAME  (2008) 

MchipnCh«*s _ 


16  August  2012 


UNCLASSIFIED 


GVSETS 


Validation  data 


Temperature  History  for  Configuration  3 
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T emperatu re  (°C)  T emperatu re  (°C) 


Temperature  History  for  Configuration  2 
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Time  (s) 
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Validation  studies 
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Validation  Techniques 

L  ^  J 

L  V  J 

Graphical 

Comparison 


Feature- 

based 

Techniques 
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PDF/CDF  - 

based 

Techniques 


Ferson  et  al 
Rutherfon 


Hypothesis 

Testing 
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Hills  and  Dowding 


Bayesian 

Posterior 

Estimation 


Liu  et  al. 
igdon  et  al. 


Classical 

Bayesian 

Bayesian 

Hypothesis 

Point-null 

Interval- 

Testing 

Hypothesis 

Based 

Brandyberry 

Testing 

Hypothesis 

McFarland  and  Mahadevan 

Testing 
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Validation  assessment 


MODELING  rnd  SIMULATION,  TESTING  rnd  VALIDATION 
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Norm-based  Norm-based  Variability-based  Variability-based 

Bayesian  Bootstrap  Bayesian  Bootstrap 


Validation  Approaches 


Liu  et  al. 


Ferson  et  al. 


Higdon  et  al. 


Hills  and  Dowding 


McFarland  and 
Mahadevan 


Brandyberry 


Rutherford 


Negligible 
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Small 
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Reject 

Accept 
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•  Statistical  power  is  the  probability  of  rejecting  the  null 
hypothesis  when  it  is  correct 

•  Size  of  a  statistical  sample  is  the  number  of  data  points 
in  a  time  series 

•  e  is  the  interval  threshold  used  in  Bayesian  hypothesis 
testing 

•  |n  is  the  population  mean  in  the  hypothesis  testing  H0: 
M<  e;  Ha:  j  ji|>  8 
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Factors  that  influence 
statistical  power 


MODELING  rno  SIMULATION,  TESTING  rno  VALIDATION 


Sample  size 

Distance  between  population  mean  and  epsilon 
Hypothesis  testing  type 


Influence  of  the  Sample  Size  and  Population  Mean 


Population  mean,  ja 


- Bayesian,  2  samples 

Classical, 2  samples 

—  —  Bayesian,  23  samples 

—  —  Classical,  23  samples 

Bayesian,  200  samples 
■  Classical,  200  samples 
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Comparison  of  statistical  power 
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Norm-based  Bootstrap  Classical  Hypothesis  Testing 

Validation  Approaches 
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Energy  and  Power 

Community  of  Interest  (E/P  Col)  MODELING  rhq  SIMULRTION,  TESTING  rhd  VRLIDRTION 


W 


Energy  and  Power  Technology  Community  of  Interest  (E/P 
Col)  formed  2009 

-  Couple  technology  to  warfighter  opportunity  areas 

-  Focus  on  developing  and  evaluating  methods,  tools  and  best 
practice  guidelines  for: 

•  Model  development,  model  sharing  and  documentation 

•  Verification,  Validation  and  Accreditation  (VV&A) 


-  Members  include: 

•  Air  Force  Research  Laboratory 

•  Electric  Ship  Research  and  Development  Consortium  (Florida  State  University) 

•  Automotive  Research  Center  (University  of  Michigan) 
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Battery  model 


MODELING  rho  SIMULATION,  TESTING  rno  VALIDATION 
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•  Testbed  for  E/P  Col 
activities:  straw- 
man  model 

•  Battery  model 
incorporated 

•  First  step  toward 
validating  the  straw- 
man  model:  validate 
the  battery  model 

•  Ultimately:  provide 
validation  metric, 
guidelines  and  tool 

X.  Lin  et  al,  “Online  Parameterization  of  Lumped  Thermal  Dynamics  in  Cylindrical  Lithium  Ion  Batteries  for  Core  Temperature  Estimation  and  Health  Monitoring”, 
IEEE  Transactions  on  Control  System  Technology,  under  review 
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Validation  results 


MODELING  rhq  SIMULRTION,  TESTING  rhd  VRLIDRTION  'V;. 


Single  Cell  -  ElectroThermal  Model 


•  Inputs 

-  I  (Current) 

-  Tf  (Inlet  Air  Temp) 

-  Vin  (Inlet  Flow  Velocity) 

•  Outputs 

-  Tc  (Core  Temperature  (°C)) 

-  Ts  (Surface  Temperature  (°C)) 

-  Vt  (Terminal  Voltage) _ 

-  SOC  (State  of  Charge) 


Thermal  Model 

-  Two  States  (Tc,  Ts) 

-  Two  Inputs  (Tf,  Vin) 


Electrical  Model:  OCV-R-RC 

-  Three  States  (SOC,  VI,  V2) 

-  One  Input  (I) 


R,  Rj 


[3]  K.  Smith,  et  at.  2010 

[4]  Y.  Hu.  et  al.  2011 


Model  Confidence  is  high  (99%) 


Data  Fragment  for  Surface  T emperature 
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Backup 
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Confidence  quantification 


Hypothesis  test: 

H0:|ji/|  <  e  (accept)  versus  Ha:|/i|  >  e  (reject) 


rd:  mean  of  rd 

n:  number  of  observations 
o2:  variance  of  rd 


rd|jU  ~  A/(jU,Z) 
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